Papers with vanilla self-attention module
Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |